1. Technical Field
The present invention relates generally to techniques for highly available, reliable, and persistent data storage in a distributed computer network.
2. Description of the Related Art
A need has developed for the archival storage of “fixed content” in a highly available, reliable and persistent manner that replaces or supplements traditional tape and optical storage solutions. The term “fixed content” typically refers to any type of digital information that is expected to be retained without change for reference or other purposes. Examples of such fixed content include, among many others, e-mail, documents, diagnostic images, check images, voice recordings, film and video, and the like. The traditional Redundant Array of Independent Nodes (RAIN) storage approach has emerged as the architecture of choice for creating large online archives for the storage of such fixed content information assets. By allowing nodes to join and exit from a cluster as needed, RAIN architectures insulate a storage cluster from the failure of one or more nodes. By replicating data on multiple nodes, RAIN-type archives can automatically compensate for node failure or removal. Typically, RAIN systems are largely delivered as hardware appliances designed from identical components within a closed system.
Managing technological obsolescence of an archive infrastructure is a key problem in preserving digital content. Given the fast pace of technological change, it is questionable whether the devices that are available today for reading tape or optical disk will still be around in ten years time. Obsolescence occurs at many levels including, without limitation, the file's original format, the application that wrote the file, and the media on which the content was recorded. At first glance, building a large scale archive that keeps pace with the latest technology while also offering online access at a reasonable cost would appear to be impossible. The problem is exacerbated because the archive must handle the complete life cycle of the data it stores. An archive that supports very long retention periods can accumulate a great deal of data over time. Archive systems must therefore be able to grow smoothly, e.g., from a few terabytes to several petabytes, and they must provide high availability, avoid data loss, and be easy to manage.
Prior art techniques do not adequately address these concerns. In particular, it is well-known that setting up large archive systems is error-prone. Databases, networking, operating systems, storage management systems and Web servers all require teams of experts with a myriad of skills to get them running together. In addition, storage systems, databases, Web servers and operating systems all have a vast range of tunable parameters that enable an administrator to optimize performance. Further, serious problems with large-scale infrastructure can take weeks to diagnose and fix. Because archive systems must be continuously available, administrators must be able to remove and replace a faulty device without interrupting ongoing service. Finally, despite the existence of various security mechanisms, administrators are mostly on their own to decide how to protect an archive from malicious attacks or inadvertent damage.
These and other problems are addressed by the present invention.